Maturity determination device and maturity determination method

ABSTRACT

A maturity determination device includes an image capturing device including a plurality of pixels arrayed one-dimensionally or two-dimensionally, the image capturing device performing image capturing of at least a part of a fruit or vegetable product to acquire an image, the plurality of pixels including a plurality of first pixels each including a first light transmission filter selectively transmitting light of a first wavelength band, the intensity of the light of the first wavelength band reflected by the fruit or vegetable product varying in accordance with a maturity level; and a signal processing circuit configured to find an area size ratio of an intensity distribution of the light of the first wavelength band on the basis of a predetermined reference value based on a pixel value obtained from the plurality of first pixels, and to generate maturity determination information in accordance with the area size ratio.

This application claims priority to Malaysian Patent Application No. PI2016001263, filed on Jul. 5, 2016, the disclosure of which is herebyincorporated by reference in its entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a maturity determination device and amaturity determination method.

2. Description of the Related Art

The economic value of a harvesting target of fruits and vegetablesdepends on the content of a specific ingredient contained in theharvesting target. For example, an oil palm bunch (Fresh Fruit Bunch)includes a large number of (e.g., 1,000 to 3,000) fruits, each of whichincludes pericarp (also referred to as “pulp”), “mesocarp” and a seed(also referred to as “palm kernel”). In the case where the harvestingtarget is an oil palm bunch, the content of oil contained in themesocarp or palm kernel in a fruit is considered important. The oilcontained in the mesocarp is generally called “palm oil”, and the oilcontained in the palm kernel is generally called “palm kernel oil”. Theoil content of the oil palm fruit is known to be different in accordancewith the maturity level of the oil palm bunch (specifically, fruits).Specifically, the oil content of a ripe fruit is known to be higher thanthe oil content of an unripe, underripe (slightly unripe) or overripefruit. For example, the oil content of a fruit when being ripe is aboutthree times the oil content thereof when being unripe.

For example, in Malaysia, the Malaysian Palm Oil Board (MPOB) hasestablished guidelines on the maturity level of fruits. Currently,farmers working in oil palm plantations subjectively determine whetheror not to harvest the fruits in accordance with the guidelines. Asdescribed above, the maturity level of the oil palm bunch significantlyinfluences the amount of yield of oil. Therefore, a wrong subjectivedetermination of a farmer (worker) on whether or not to harvest thefruits may have a significant influence on the amount of yield. Inactuality, it is difficult for a farmer to correctly determine whetherthe fruits are ripe or not, which is a factor that decreases the amountof yield of oil.

In such a situation, measurement equipments have been developed formeasuring various characteristics of an oil palm bunch, which is asubject (harvesting target), by use of near infrared light withoutdestroying the bunch, namely, in a nondestructive manner. Suchmeasurement equipments irradiate the subject with near infrared light ofa predetermined wavelength and measure a characteristic to be measuredbased on the reflectance of the light from the subject.

WO2012/074372 discloses a system that determines the maturity level ofoil palm fruits using a hyperspectral imaging technology. WO2012/074372also discloses the characteristic that the reflectance of the light ofthe near infrared wavelength band varies in accordance with the maturitylevel of the fruit. In this system, an image of a harvested oil palmbunch is captured by a spectral camera to acquire a sample image, andthe sample image is analyzed based on the characteristic that thereflectance varies in accordance with the maturity level of the fruitsto determine the maturity level of the oil palm fruit.

Japanese Laid-Open Patent Publication No. 2000-356591 discloses anondestructive sugar content meter that calculates the sugar content offruits and vegetables by a sugar content estimation equation by use ofan output from a photoelectric conversion element based on light thathas passed a near infrared light wavelength filter. Japanese Laid-OpenPatent Publication No. Hei 8-122250 discloses a nondestructivemeasurement device that determines the maturity level of fruits andvegetables based on the intensity ratio between two specific wavelengthcomponents of near infrared light.

SUMMARY OF THE INVENTION

As described above, the economic value of a harvesting target depends onthe content of a specific ingredient contained in the harvesting target.Therefore, it is desired to harvest the harvesting target at the timewhen the content of the specific component in the harvesting target ishigh. For example, in the case where the harvesting target is an oilpalm bunch, it is desired to harvest the oil palm bunch at the time whenthe content of the palm oil or palm kernel oil is high.

However, the system disclosed in WO2012/074372 is developed for indooruse, and outdoor use is not fully considered. Therefore, the system isof a large scale. In addition, the hyperspectral camera is costly, andthus the system is also costly. In a high-temperature, high-humidityenvironment, the hyperspectral camera is not considered durable.Therefore, it is very difficult to perform, for example, image capturingof an oil palm bunch in a tree in an oil palm plantation by use of thehyperspectral camera and determine the maturity level on the spot inorder to determine an appropriate time to harvest the oil palm bunch. Itis now assumed that the time to harvest is determined on the spot by useof the hyperspectral camera and an oil palm bunch having a high economicvalue is harvested. However, even in such a case, the load on the farmeris increased by the large-scale system, which lowers the efficiency ofthe entire harvesting work. As a result, the producer does not fullyenjoy the advantage that the economic value of the harvesting target isimproved.

The present invention made to solve the above-described problem providesa maturity determination device and a maturity determination methoddetermining, with high precision, the time to harvest a harvestingtarget while reducing the load on a farmer without decreasing theefficiency of the harvesting work.

A maturity determination device in an embodiment according to thepresent invention is a maturity determination device determining amaturity level of a fruit or vegetable product, and includes an imagecapturing device including a plurality of pixels arrayedone-dimensionally or two-dimensionally, the image capturing deviceperforming image capturing of at least a part of the fruit or vegetableproduct to acquire an image, the plurality of pixels including aplurality of first pixels each including a first light transmissionfilter selectively transmitting light of a first wavelength band, theintensity of the light of the first wavelength band reflected by thefruit or vegetable product varying in accordance with the maturitylevel; and a signal processing circuit configured to find an area sizeratio of an intensity distribution of the light of the first wavelengthband on the basis of a predetermined reference value based on pixelvalues obtained from the plurality of first pixels, and to generatematurity determination information in accordance with the area sizeratio.

In an embodiment, the first wavelength band may be a wavelength band ofnear infrared light.

In an embodiment, the first wavelength band may be a wavelength band ofnear infrared light from 800 nm to 900 nm.

In an embodiment, the signal processing circuit may calculate the numberof pixels having a pixel value larger than, or equal to, a firstthreshold value or a pixel value smaller than, or equal to, the firstthreshold value from the plurality of first pixels, may find a firstratio of the calculated number of the pixels with respect to the numberof the plurality of first pixels, and may generate the maturitydetermination information based on a result of comparing the first ratioagainst a second threshold value.

In an embodiment, the plurality of pixels may further include aplurality of second pixels each including a second light transmissionfilter selectively transmitting light of a second wavelength band, theintensity of the light of the second wavelength band reflected by thefruit or vegetable product varying in accordance with the maturitylevel, the second wavelength band being different from the firstwavelength band; and the signal processing circuit may generate thematurity determination information based on the pixel values obtainedfrom the plurality of first pixels and pixel values obtained from theplurality of second pixels.

In an embodiment, the second wavelength band may be a wavelength band ofred light.

In an embodiment, the signal processing circuit may add a pixel valueobtained from each first pixel of the plurality of first pixels and apixel value obtained from each second pixel of the plurality of secondpixels on a pixel-by-pixel basis, said each second pixel beingassociated with said each first pixel, may calculate the number ofpixels having a sum value larger than, or equal to, a third thresholdvalue or a sum value smaller than, or equal to, the third thresholdvalue from pixels as targets of addition, may find a second ratio of thecalculated number of the pixels with respect to the number of the pixelsas the targets of addition, and may generate the maturity determinationinformation based on a result of comparing the second ratio against afourth threshold value.

In an embodiment, the signal processing circuit may subtract a pixelvalue obtained from each second pixel of the plurality of second pixelsfrom a pixel value obtained from each first pixel of the plurality offirst pixels on a pixel-by-pixel basis, said each second pixel beingassociated with said each first pixel, may calculate the number ofpixels having a difference value larger than, or equal to, a thirdthreshold value or a difference value smaller than, or equal to, thethird threshold value from pixels as targets of subtraction, may find athird ratio of the calculated number of the pixels with respect to thenumber of the pixels as the targets of subtraction, and may generate thematurity determination information based on a result of comparing thethird ratio against a fourth threshold value.

In an embodiment, the signal processing circuit may calculate the numberof pixels having a pixel value larger than, or equal to, a fifththreshold value or a pixel value smaller than, or equal to, the fifththreshold value from the plurality of second pixels, may find a fourthratio of the calculated number of the pixels with respect to the numberof the plurality of second pixels, and may generate the maturitydetermination information based on a result of comparing the first ratioagainst the second threshold value and a result of comparing the fourthratio against a sixth threshold value.

In an embodiment, the plurality of pixels further may include aplurality of third pixels each including a third light transmissionfilter selectively transmitting blue light and a plurality of fourthpixels each including a fourth light transmission filter selectivelytransmitting green light; and the signal processing circuit may generatea color image based on pixel values obtained from the plurality ofsecond, third and fourth pixels, and may generate, based on the maturitydetermination information, a maturity level image including informationon the reference value and representing the maturity level.

In an embodiment, the maturity determination device may further includean output interface which outputs the maturity determination informationto outside.

In an embodiment, the maturity determination device may further includea driving circuit which generates a driving signal for driving anotification device notifying the maturity level, the notificationdevice being connectable with the maturity determination device, and thedriving signal being generated in accordance with the maturitydetermination information.

In an embodiment, the maturity determination device may further includea notification device which notifies the maturity level; and a drivingcircuit which generates a driving signal for driving the notificationdevice, the driving signal being generated in accordance with thematurity determination information.

In an embodiment, the notification device may include at least one of anoptical device emitting light in accordance with the driving signal, asound output device outputting a sound in accordance with the drivingsignal, a vibration device vibrating in accordance with the drivingsignal, and a display device displaying maturity level information inaccordance with the driving signal.

In an embodiment, the maturity determination device may further includea display device; and a driving circuit which generates a driving signalfor driving the display device, the driving signal being generated inaccordance with the maturity determination information. The displaydevice may display the maturity level image as overlapping the colorimage in accordance with the driving signal.

In an embodiment, the signal processing circuit may determine whetherthe fruit or vegetable product is harvestable or not based on thematurity determination information.

In an embodiment, the light reflected by the fruit or vegetable productat the time of harvest thereof may have a reflectance characteristicthat an intensity thereof increases as the light has a longer wavelengthin a wavelength band from the blue light to the near infrared light.

In an embodiment, the fruit or vegetable product may be a bunch with alarge number of fruits.

A method in an embodiment according to the present invention is a methodfor determining a maturity level of a fruit or vegetable product, andincludes the steps of receiving an image including at least a part ofthe fruit or vegetable product, the image including an intensitydistribution of light of at least a first wavelength band, the intensityof the light of the first wavelength band reflected by the fruit orvegetable product varying in accordance with the maturity level; findingan area size ratio of the intensity distribution of the light of thefirst wavelength band on the basis of a predetermined reference value;and generating maturity determination information in accordance with thearea size ratio.

In an embodiment, the first wavelength band may be a wavelength band ofnear infrared light.

In an embodiment, the first wavelength band may be a wavelength band ofnear infrared light from 800 nm to 900 nm.

In an embodiment, in the step of finding the area size ratio, the numberof pixels having a pixel value larger than, or equal to, a firstthreshold value or a pixel value smaller than, or equal to, the firstthreshold value may be calculated from the plurality of first pixelsincluding information, representing the intensity distribution of thelight of the first wavelength band, and included in the image; and afirst ratio of the calculated number of the pixels with respect to thenumber of the plurality of first pixels may be found. In the step ofgenerating the maturity determination information, the maturitydetermination information may be generated in accordance with a resultof comparing the first ratio against a second threshold value.

In an embodiment, in the step of receiving the image, the image receivedmay further include an intensity distribution of light of a secondwavelength band, the intensity of the light of the second wavelengthband reflected by the fruit or vegetable product varying in accordancewith the maturity level, the second wavelength band being different fromthe first wavelength band. In the step of generating the maturitydetermination information, the maturity determination information may begenerated based on the intensity distributions of the light of the firstwavelength band and the light of the second wavelength band.

In an embodiment, the second wavelength band may be a wavelength band ofred light.

In an embodiment, in the step of finding the area size ratio, a pixelvalue of each first pixel of the plurality of first pixels includinginformation, representing the intensity distribution of the light of thefirst wavelength band, and included in the image, and a pixel value ofeach second pixel of the plurality of second pixels includinginformation, representing the intensity distribution of the light of thesecond wavelength band, and included in the image, may be added togetheron a pixel-by-pixel basis, said each second pixel being associated withsaid each first pixel; the number of pixels having a sum value largerthan, or equal to, a third threshold value or a sum value smaller than,or equal to, the third threshold value may be calculated from pixels astargets of addition; and a second ratio of the calculated number of thepixels with respect to the number of the pixels as the targets ofaddition may be found. In the step of generating the maturitydetermination information, the maturity determination information may begenerated in accordance with a result of comparing the second ratioagainst a fourth threshold value.

In an embodiment, in the step of finding the area size ratio, a pixelvalue of each second pixel of the plurality of second pixels includinginformation, representing the intensity distribution of the light of thesecond wavelength band, and included in the image may be subtracted, ona pixel-by-pixel basis, from a pixel value of each first pixel of theplurality of first pixels including information, representing theintensity distribution of the light of the first wavelength band, andincluded in the image, said each second pixel being associated with saideach first pixel; the number of pixels having a difference value largerthan, or equal to, a third threshold value or a difference value smallerthan, or equal to, the third threshold value may be calculated frompixels as targets of subtraction; and a third ratio of the calculatednumber of the pixels with respect to the number of the pixels as thetargets of subtraction may be found. In the step of generating thematurity determination information, the maturity determinationinformation may be generated in accordance with a result of comparingthe third ratio against a fourth threshold value.

In an embodiment, in the step of finding the area size ratio, the numberof pixels having a pixel value larger than, or equal to, a fifththreshold value or a pixel value smaller than, or equal to, the fifththreshold value may be calculated from the plurality of second pixelsincluding information, representing the intensity distribution of thelight of the second wavelength band, and included in the image; a fourthratio of the calculated number of the pixels with respect to the numberof the plurality of second pixels may be found; and the maturitydetermination information may be generated in accordance with a resultof comparing the first ratio against the second threshold value and aresult of comparing the fourth ratio against a sixth threshold value.

In an embodiment, in the step of receiving the image, the image receivedmay further include intensity distributions of blue light and greenlight. The method may further include the step of generating a colorimage based on pixel values obtained from the plurality of pixelsincluding information representing the intensity distributions of thered light, the blue light and the green light, and generating a maturitylevel image including information on the reference value andrepresenting the maturity level, the maturity level image beinggenerated based on the maturity determination information.

In an embodiment, the method may further include the step of outputtingthe maturity determination information to outside.

In an embodiment, the method may further include the step of generatinga driving signal for driving a notification device notifying thematurity level, the driving signal being generated in accordance withthe maturity determination information.

In an embodiment, the notification device may include at least one of anoptical device emitting light in accordance with the driving signal, asound output device outputting a sound in accordance with the drivingsignal, a vibration device vibrating in accordance with the drivingsignal, and a display device displaying maturity level information inaccordance with the driving signal.

In an embodiment, the method may further include the step of displayingthe maturity level image as overlapping the color image on a displaydevice.

In an embodiment, the method may further include the step of performingimage capturing of at least a part of the fruit or vegetable product toacquire the image.

In an embodiment, in the step of acquiring the image, the image may beacquired by performing image capturing of a central part and thevicinity thereof of the fruit or vegetable product.

In an embodiment, in the step of acquiring the image, the image may beacquired a plurality of times.

In an embodiment, the method may further include the step of determiningwhether the fruit or vegetable product is harvestable or not based onthe maturity determination information.

In an embodiment, the light reflected by the fruit or vegetable productat the time of harvest thereof may have a reflectance characteristicthat an intensity thereof increases as the light has a longer wavelengthin a wavelength band from the blue light to the near infrared light.

In an embodiment, the fruit or vegetable product may be a bunch with alarge number of fruits.

A computer program in an embodiment according to the present inventioncauses a computer, usable for a maturity determination devicedetermining a maturity level of a fruit or vegetable product, to executethe steps of receiving an image including at least a part of the fruitor vegetable product, the image including an intensity distribution oflight of at least a first wavelength band, the intensity of the light ofthe first wavelength band reflected by the fruit or vegetable productvarying in accordance with the maturity level; finding an area sizeratio of the intensity distribution of the light of the first wavelengthband on the basis of a predetermined reference value; and generatingmaturity determination information in accordance with the area sizeratio.

According to an embodiment of the present invention, a maturitydetermination device and a maturity determination method determining,with high precision, the time to harvest a harvesting target whilereducing the load on a farmer without decreasing the efficiency of theharvesting work are provided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a structural view of an example of system 1000 with which adetermination method in an embodiment is usable.

FIG. 2 is a schematic view of pixels of a pixel unit arrayed in fourrows by four columns among a plurality of pixels 221 arrayedtwo-dimensionally in a pixel array 220A.

FIG. 3 is a schematic view showing a cross-section of the pixel 221 thatis parallel to an optical axis of a microlens 222.

FIG. 4 is a graph showing the light transmission characteristic of alight transmission filter 223.

FIG. 5 is a schematic view showing a structure of another pixel unit inthe pixel array 220A.

FIG. 6 is a flowchart showing an example of processing procedure of thedetermination method in the embodiment.

FIG. 7 is a graph schematically showing how the intensity of thereflected light (reflectance) varies in accordance with the maturitylevel at the time of harvest of a bunch F.

FIG. 8 shows images I acquired by image capturing of the entirety of thebunches F of three maturity levels (unripe, underripe, and ripe), andpixel distributions, at the respective maturity levels, of pixels havinga pixel value larger than, or equal to, a reference value among thepixels that have a pixel value NIRS and are included in the images I.

FIG. 9A shows an image I of the unripe bunch F and a mapping imageobtained as a result of mapping performed based on the pixel value NIRSof a plurality of first pixels in the image I.

FIG. 9B shows an image I of the underripe bunch F and a mapping imageobtained as a result of mapping performed based on the pixel value NIRSof a plurality of first pixels in the image I.

FIG. 9C shows an image I of the ripe bunch F and a mapping imageobtained as a result of mapping performed based on the pixel value NIRSof a plurality of first pixels in the image I.

FIG. 10 shows color mapping images obtained as a result of dividing eachof three mapping images of the unripe, underripe and ripe bunches F intothree hierarchical layers I, II and III.

FIG. 11 is a graph obtained as a result of performing normal probabilityplotting of a signal strength of near infrared light (i.e., pixel valueNIRS) reflected by each of the unripe, underripe and ripe bunches F.

FIG. 12 is a flowchart showing a specific processing procedure in stepS200.

FIG. 13A is a flowchart showing a specific processing procedure in stepS300.

FIG. 13B is a flowchart showing another specific processing procedure instep S300.

FIG. 14A is a graph obtained as a result of performing normalprobability plotting of a signal strength of red light (i.e., pixelvalue REDS) reflected by each of the unripe, underripe and ripe bunchesF.

FIG. 14B is a graph obtained as a result of performing normalprobability plotting of the pixel value that is obtained as a result ofadding the pixel value NIRS and the pixel value REDS.

FIG. 15 shows color mapping images obtained as a result of dividing eachof three mapping images of the unripe, underripe and ripe bunches F intothree hierarchical layers I, II and III.

FIG. 16 is a flowchart showing a specific processing procedure in stepS200.

FIG. 17 is a graph obtained as a result of performing normal probabilityplotting of the pixel value that is obtained as a result of subtractingthe pixel value REDS from the pixel value NIRS.

FIG. 18 shows color mapping images obtained as a result of dividing eachof three mapping images of the unripe, underripe and ripe bunches F intothree hierarchical layers I, II and III.

FIG. 19 is a flowchart showing a specific processing procedure in stepS200.

FIG. 20 is a flowchart showing a specific processing procedure in stepS200.

FIG. 21 is a block diagram schematically showing a block structure of amaturity determination device 100.

FIG. 22 is a block diagram schematically showing functional blocks of asignal processing circuit 300.

FIG. 23 is a block diagram schematically showing a block structure of amaturity determination device 100A in a variation of the embodiment.

FIG. 24 is a block diagram schematically showing a block structure of amaturity determination device 100B in another variation of theembodiment.

FIG. 25A is a graph showing the wavelength dependence of the reflectanceof light reflected by green apple.

FIG. 25B is a graph showing the wavelength dependence of the reflectanceof light reflected by mango.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

As a result of accumulating studies, the present inventors found thefollowing problems.

There are problems to be solved in order to determine, highly precisely,the time to harvest of a harvesting target. Oil palm is harvested inunits of bunches each including a larger number of fruits. There is adistribution of maturity level of fruits included in one bunch, which isone factor that makes it difficult to determine the maturity level. Inthe case where one bunch, which is a unit of harvesting, includes alarge number of fruits as in the case of oil palm, the maturity levelcannot be determined by simply averaging pixel values of a plurality ofpixels included in an image of the bunch. One bunch includes dead calyxand leaves in addition to the fruits, which is also a factor that makesit difficult to determine the maturity level. The image of the bunch mayinclude information on the dead calyx and the leaves, which is notrelated to the fruits. Therefore, it is difficult to use only theinformation on the fruits to determine the maturity level.

Based on the above-described knowledge, the present inventors found amethod by which the maturity level is determined highly precisely evenin the case where a bunch includes a large number of fruits havingdifferent maturity levels and also includes dead calyx and leaves as inthe case of oil palm, and thus have arrived at the present invention.

A maturity determination device in an embodiment according to thepresent invention determines the maturity level of a fruit or vegetableproduct, and is preferably usable as a device determining the maturitylevel of, for example, an oil palm bunch including a large number offruits. The maturity determination device includes an image capturingdevice including a plurality of pixels arrayed one-dimensionally ortwo-dimensionally, the image capturing device performing image capturingof at least a part of the fruit or vegetable product to acquire animage, the plurality of pixels including a plurality of first pixelseach including a first light transmission filter selectivelytransmitting light of a first wavelength band, the intensity of thelight of the first wavelength band reflected by the bunch varying inaccordance with the maturity level; and a signal processing circuitconfigured to find an area size ratio of an intensity distribution ofthe light of the first wavelength band on the basis of a predeterminedreference value based on a pixel value obtained from the plurality offirst pixels, and to generate maturity determination information inaccordance with the area size ratio. The light of the first wavelengthband is typically near infrared light. With this structure, the maturitylevel of a fruit or vegetable product is determined with high precision.

A method in an embodiment according to the present invention is a methoddetermining the maturity level of a fruit or vegetable product, and ispreferably usable as a method determining the maturity level of, forexample, an oil palm bunch including a large number of fruits. Themethod includes the steps of receiving an image including at least apart of the fruit or vegetable product, the image including an intensitydistribution of light of at least a first wavelength band, the intensityof the light of the first wavelength band reflected by the bunch varyingin accordance with the maturity level; finding an area size ratio of theintensity distribution of the light of the first wavelength band on thebasis of a predetermined reference value; and generating maturitydetermination information in accordance with the area size ratio. Withthis method, the maturity level of a fruit or vegetable product isdetermined with high precision.

In this specification, a maturity determination device and a maturitydetermination method determining the maturity level of an oil palmbunch, which is an example of fruit or vegetable product. In thisspecification, any type or any part of fruit or vegetable that is to beharvested is referred to as a “fruit or vegetable product”. As describedbelow, the present invention encompasses determination of the maturitylevel of a fruit or vegetable product with which the reflectance oflight varies in accordance with the maturity level in a specificwavelength band, for example, coffee fruits, apples and mangoes.

Hereinafter, a maturity determination device and a maturitydetermination method in embodiments according to the present inventionwill be described with reference to the attached drawings. In thefollowing description, identical or similar elements will bear theidentical reference signs to each other. The maturity determinationdevice and the maturity determination method in an embodiment accordingto the present invention are not limited to those in any of thefollowing embodiments. For example, one embodiment and anotherembodiment may be combined together.

Embodiment 1

With reference to FIG. 1 through FIG. 13B, a method for determining thematurity level of an oil palm bunch F (hereinafter, referred to as a“determination method”) in this embodiment will be described.

[Structure of System 1000 for which the Determination Method is Usable]

An example of structure of a system 1000 for which the determinationmethod in this embodiment is usable will be described. The determinationmethod in this embodiment may also be preferably usable for anotherdevice (or a system) different from the system 1000. A specific exampleof the another device (in this specification, will be referred to as a“maturity determination device”) will be described below in detail.

FIG. 1 shows an example of structure of the system 1000 for which thedetermination method in this embodiment is usable. The system 1000typically includes an image capturing device 200, a calculation device Pand a notification device 500. In the system 1000, the image capturingdevice 200 performs image capturing of at least a part of a large numberof fruits in an oil palm bunch (hereinafter, referred to simply as the“bunch”) F to acquire an image of the fruits. In this case, it ispreferable that the image is of at least three fruits. The calculationdevice P analyzes the image to generate maturity determinationinformation indicating the maturity level of the bunch F. For example,the notification device 500 notifies a farmer (worker) of whether thebunch F is ripe or not in accordance with the maturity determinationinformation.

The image capturing device 200 includes a lens 210 and an image sensor220. The image capturing device 200 may be realized as, for example, acamera module.

The lens (or lens group) 210 collects light from the bunch F onto animage capturing surface of the image sensor 220. The lens 210 may be asingle lens or include a plurality of lenses. The lens 210 may include alens for autofocus (AF) and/or a lens for optical zooming. The lens forAF and the lens for optical zooming are drivable by a dedicated driver(not shown). The image capturing device 200 may include a controlcircuit (not shown) controlling such a driver.

The image sensor 220 is, for example, a CCD (Charge Coupled Device)sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. Theimage sensor 220 includes a plurality of pixels arrayedone-dimensionally or two-dimensionally (pixel array 220A shown in FIG.2). In the case where the plurality of pixels are arrayedone-dimensionally, the image sensor 220 may be a line sensor.

FIG. 2 shows a part of a plurality of pixels 221 arrayedtwo-dimensionally in the pixel array 220A, more specifically, the pixels221 arrayed in four rows by four columns. FIG. 3 schematically shows across-section of the pixel 221 that is parallel to an optical axis of amicrolens 222. FIG. 4 shows a light transmission characteristic of alight transmission filter 223. In the graph of FIG. 4, the horizontalaxis represents the wavelength (nm) of the light, and the vertical axisrepresents the transmittance of the light transmission filter.

The image sensor 220 includes the plurality of pixels 221 arrayedtwo-dimensionally (pixel array 220A). The pixels 221 each include themicrolens 222, the light transmission filter 223 and a photoelectricconversion element 224.

The microlens 222 is located on the light transmission filter 223, andcollects light from the bunch F to improve the pixel sensitivity. Thelight transmission filter 223 selectively transmits light of a specificwavelength band. For example, the light transmission filter 223 iseither an IR filter or one of RGB color filters. As shown in FIG. 4, theIR filter selectively transmits light of a near infrared wavelength band(e.g., 800 nm to 2500 nm), and preferably selectively transmits light ofa wavelength band of 800 nm to 900 nm. An R filter selectively transmitslight of a red wavelength band (e.g., 620 nm to 750 nm). A G filterselectively transmits light of a green wavelength band (e.g., 500 nm to570 nm). A blue filter selectively transmits light of a blue wavelengthband (e.g., 450 nm to 500 nm).

The photoelectric conversion element 224 is typically a photodiode (PD),and converts received light into an electric signal. The PD is, forexample, embedded in a semiconductor substrate formed of silicon (notshown).

As shown in FIG. 2, the pixel array 220A has pixel units each includingpixels 221A, 221B, 221C and 221D arrayed in two rows by two columns. Thepixels 221A, 221B, 221C and 221D respectively include an R filter, a Gfilter, an IR filter and a B filter. Such an array corresponds to anarray obtained as a result of a G filter in an odd or even number columnof a Bayer array being replaced with an IR filter. In thisspecification, the pixels 221A, 221B, 221C and 221D included in onepixel unit are associated with each other. As described below, pixelvalues of the pixels associated with each other (e.g., pixels 221A and221C) may be subjected to addition or subtraction.

FIG. 5 shows a structure of other pixel units in the pixel array 220A.FIG. 5 shows a part of the plurality of pixels 221 arrayedtwo-dimensionally in the pixel array 220A, more specifically, the pixels221 arrayed in eight rows by eight columns. The pixel array 220A mayinclude pixel units each including the pixels 221 arrayed in four rowsby four columns. In this example, in each pixel unit, the number of theG filters is smallest and the number of the IR filters is largest. Areason for this is that in the determination method in this embodiment,the pixel value obtained from the pixel 221C including the IR filter isconsidered most important and the pixel value obtained from the pixel221B including the G filter is not used.

Any of various pixel unit patterns other than the above-described pixelunit patterns may be selected. For example, the pixel unit does not needto include the B filter, or may include only the IR filter and the Rfilter. The number of each type of filters included in the pixel unit isarbitrary.

As described below in detail, the reflectance of light (intensity ofreflected light) of a first wavelength band from, for example, the bunchF varies in accordance with the maturity level. The first wavelengthband is, for example, a near infrared wavelength band. In the case wherethe maturity level of a fruit or vegetable product exhibiting such acharacteristic is to be found, the unit pixel merely needs to include atleast the IR filter. For example, the four filters in the pixel unitshown in FIG. 2 may all the IR filters. Even in such a case, thedetermination method in this embodiment is usable. However, as describedbelow, it is preferable that the unit pixel includes the RGB filtersfrom the point of view of generating a color image.

In this specification, the term “pixel value” is, for example, an 8-bitgray scale value, and mainly refers to RAW data. The image capturingdevice 200 at least outputs RAW data. Alternatively, the image capturingdevice 200 may have a function of generating a luminance/colordifference signal based on the RAW data and outputting theluminance/color difference signal.

The calculation device P includes a signal processing circuit (notshown) that processes pixel values output from the image capturingdevice 200 to generate maturity determination information. Thecalculation device P is, for example, a personal computer (PC) such as alaptop computer or the like. Data transmission between the calculationdevice P and the image capturing device 200 may be performed by, forexample, wireless communication. The wireless communication may beconformed to, for example, the Bluetooth (registered trademark)standards. Needless to say, the calculation device P and the imagecapturing device 200 may be connected with each other in a wired mannerby, for example, a USB cable or the like. The calculation device P mayfurther include a driving circuit (not shown) that generates a drivingsignal for driving the notification device 500 in accordance with thematurity determination information. All of, or a part of, thesefunctions may be implemented on the image capturing device 200.

The calculation device P may have a function of performing a processgenerally performed for image processing, for example, gamma correction,color interpolation, spatial interpolation, and auto-white balance.

The notification device 500 notifies the farmer of the maturity level ofthe bunch F or whether the bunch F is harvestable or not. Thenotification device 500 is connected with the calculation device P in awired or wireless manner. The notification device 500 includes at leastone of an optical device emitting light in accordance with a drivingsignal, a sound output device outputting a sound in accordance with thedriving signal, a vibration device vibrating in accordance with thedriving signal, and a display device displaying maturity information inaccordance with the driving signal. The optical device is, for example,an LED (Light Emitting Diode) lamp. The sound output device is, forexample, a speaker. The vibration device is, for example, a vibrator.The display device is, for example, a liquid crystal display or anorganic EL (Electroluminescence) display.

[Processing Procedure of the Determination Method]

FIG. 6 shows an example of processing procedure of the determinationmethod in this embodiment. Hereinafter, a method for determining thematurity level of the bunch F in accordance with the processingprocedure shown in FIG. 6 by use of the system 1000 will be described.The operating subject executing each of steps is the calculation deviceP, specifically, a signal processing circuit or a driving circuit of thecalculation device P.

(Step S100)

The image capturing device 200 performs image capturing of all of, or apart of, the large number of fruits in the bunch F (preferably, at leastthree fruits) to acquire an image I. As described below, the imagecapturing device 200 may perform image capturing of the entirety of theoil palm tree. The image capturing device 200 may acquire the image I byperforming image capturing of a central part and the vicinity thereof ofthe bunch F. Moreover, the image capturing device 200 preferablyacquires the image I through image capturing of a well-sunlit portion ofthe bunch F, e.g., a tip thereof. The reason is that, presumably, thefruits in a sunlit portion of the bunch F will be the first to startmaturing. The image capturing device 200 may perform the image capturinga plurality of times to acquire the image I a plurality of times. Theacquired pieces of image data may be added together to improve the SNratio. The acquired image I includes an intensity distribution of atleast near infrared light reflected by the bunch F. In this embodiment,the image I includes intensity distributions of the near infrared light,red light, green light and blue light reflected by the bunch F. In otherwords, the image I includes pixel values (image data) REDS, GLNS, NIRSand BLUS obtained from the four pixels 221A, 221B, 221C and 221D. In theembodiments of the present invention, the image I merely needs toinclude an intensity distribution of at least the near infrared light.

In step S100, the calculation device P receives the image I from theimage capturing device 200. The operation of “receiving the image”encompasses acquiring the image I in real time when the image capturingdevice 200 acquires the image I. The operation of “receiving the image”is not limited to this. For example, the calculation device P mayacquire the image I later by reading the image I stored on a storagemedium such as an external hard disc HD or the like connected with thecalculation device P. Such an operation may be encompassed in theoperation of “receiving the image”.

FIG. 7 schematically shows how the intensity of the reflected light(reflectance) varies in accordance with the maturity level at the timeof harvest of the bunch F. The horizontal axis represents the wavelengthof the light (nm), and the vertical axis represents the reflectance. Ingeneral, the image sensor 220 (e.g., image sensor using silicon) has asensitivity characteristic depending on the wavelength of the light.Generally, for measuring the reflectance in consideration of thesensitivity characteristic, the pixel value that is output from theimage sensor 220 is standardized by use of a standard white plate. Thestandard white plate is a highly reflective diffuse reflection platethat generates diffuse reflection light not depending on the angle. Thereflectance represented by the vertical axis of FIG. 7 shows the ratioof the reflectance from the bunch F with respect to the reflectancemeasured by use of the standard white plate.

The maturity may be classified into, for example, “ripe”, “underripe”and “unripe”. Needless to say, the maturity may be classified into alarger number of categories. For example, the category “overripe” may beprovided. A category representing another maturity level (e.g.,“slightly ripe” may be provided between “ripe” and “underripe”. At thetime of harvest of the oil palm bunch F, the light reflected by thebunch F has a reflectance characteristic that the intensity increases asthe light has a longer wavelength within the wavelength band from theblue light to the near infrared light. Especially in a wavelength bandof the red light, which is visible light, the reflectance characteristicis exhibited that the intensity increases as the maturity level of thebunch F rises. Also in the near infrared light wavelength band havingthe maximum wavelength of the reflected light that may be received inaccordance with the sensitivity characteristic of the image sensor 220,the reflectance characteristic is exhibited that the intensity increasesas the maturity level of the bunch F rises. As can be seen, especiallyin the wavelength band from the red light to the near infrared light,the reflectance is represented by a curve unique to the maturity level.As described below in detail, an appropriate threshold value for thereflectance may be set in each wavelength band based on the curve andthe reflectance and the threshold value may be compared against eachother, so that the maturity level is determined.

In this embodiment according to the present invention, the image Iincluding the intensity distribution of at least the near infrared lightmay be analyzed, so that the maturity level is determined. In thisembodiment, the maturity level is determined based on the pixel valueNIRS. Therefore, for example, the pixel unit including the pixels 221 intwo rows by two columns shown FIG. 2 preferably includes the pixels 221Cincluding the IR filter in the largest number. As described below indetail, the image I may also be analyzed by use of both of the pixelvalue NIRS and the pixel value REDS. This allows the maturity level tobe determined with higher precision. In this case, the image I includesthe intensity distributions of at least the near infrared light and thered light.

(Step S200)

The calculation device P calculates an area size ratio of the intensitydistribution of the near infrared light in the received image I. Theimage I includes a plurality of first pixels (or image data) includinginformation representing the intensity distribution of the near infraredlight, a plurality of second pixels including information representingan intensity distribution of the red light, a plurality of third pixelsincluding information representing an intensity distribution of thegreen light, and a plurality of fourth pixels including informationrepresenting an intensity distribution of the blue light. As describedabove, the image I merely needs to include at least the plurality offirst pixels.

FIG. 8 shows the images I acquired by image capturing of the entirety ofthe bunches F of three maturity levels (unripe, underripe and ripe), andalso shows, for each maturity level, the distribution of the pixelshaving a pixel value higher than, or equal to, a reference value, amongthe pixels having the pixel value NIRS included in the pixel I. In thisembodiment, the pixel value is represented by, for example, the 8-bitsystem (values of 0 to 255). The reference value may be, for example,“80”. In this specification, the pixel value may be referred to also asthe “gray scale value”.

It is now assumed that all the pixel values in the image I including theentirety of the bunch F are simply averaged. The average value may varyby an influence of the background, the shadow between the fruits or thelike, which may be included in the image I. With the determinationmethod in this embodiment, the area size of the intensity distributionof the near infrared light is found on the basis of the predeterminedreference value. FIG. 8 shows, for each maturity level, how the pixelshaving a pixel value of, for example, 80 or larger, among the pixelshaving the pixel value NIRS included in the image I, are distributed inthe image I. As the maturity level of the bunch F rises, the density ofthe pixels (pixel distribution) increases. This indicates that thematurity determination information may be obtained based on the pixeldistribution on the basis of the reference value. It should be notedthat the pixel distribution still includes information on the backgroundor the like.

Portion (a) of FIG. 9A shows an image including the entirety of thebunch F that is unripe. Portion (b) of FIG. 9A shows, in an enlargedmanner, the rectangular part of the image shown in portion (a) of FIG.9A represented by the dashed line. Portion (c) of FIG. 9A is a mappingimage obtained as a result of the pixel distribution of pixels in arange of predetermined pixel values being mapped based on the pixelvalue NIRS of the plurality of first pixels included in the image Ishown in portion (b) of FIG. 9A. Portion (a) of FIG. 9B shows an imageincluding the entirety of the bunch F that is slightly underripe.Portion (b) of FIG. 9B shows, in an enlarged manner, the rectangularpart of the image shown in portion (a) of FIG. 9B represented by thedashed line. Portion (c) of FIG. 9B is a mapping image obtained as aresult of the pixel distribution of pixels in the range of predeterminedpixel values being mapped based on the pixel value NIRS of the pluralityof first pixels included in the image I shown in portion (b) of FIG. 9B.Portion (a) of FIG. 9C shows an image including the entirety of thebunch F that is ripe. Portion (b) of FIG. 9C shows, in an enlargedmanner, the rectangular part of the image shown in portion (a) of FIG.9C represented by the dashed line. Portion (c) of FIG. 9C is a mappingimage obtained as a result of the pixel distribution of pixels in therange of predetermined pixel values being mapped based on the pixelvalue NIRS of the plurality of first pixels included in the image Ishown in portion (b) of FIG. 9C.

For example, it is now assumed that the entirety of the bunch F isprocessed with image capturing from far to acquire the image I shown inportion (a) of FIG. 9A. In this case, as described above, the image Iincludes complicated information other than the bunch F, for example,the background, the shadow and the like. Such complicated informationmay possibly have adverse information on the maturity determination. Inorder to suppress this influence, it is preferable to perform imagecapturing of, for example, the rectangular part represented by thedashed line shown in portion (a) of FIG. 9A in an enlarged manner. As aresult, the image I shown in portion (b) of FIG. 9A is acquired. In thiscase, the image capturing device 200 may include a narrow angle lenshaving a relatively long focal distance, or may include an optical zoomlens. Alternatively, the image capturing device 200 may have anelectronic zoom function, needless to say. The rectangular partpreferably includes at least three fruits.

After an image including one or a plurality of bunches F or the entiretyof the oil palm tree is acquired by the image capturing device 200, thecalculation device P may extract the rectangular part represented by thedashed line from such an image by trimming. In other words, thecalculation device P may use a trimming function to acquire the image Ishown in portion (b) of FIG. 9A. In this manner, in the case where, forexample, an image of the entirety of the oil palm tree is captured fromfar, information on the fruits necessary for the maturity determinationmay be preferably obtained while the above-described complicatedinformation is eliminated. It is preferable that the image I to betrimmed also includes at least three fruits.

For performing image capturing of the bunch F or the entirety of the oilpalm tree, a part of bunch F including, for example, three fruits may beautomatically specified (recognized) by image recognition. Trimming maybe performed in substantially the same manner. An example of usableimage recognition technology may be any of a wide range of known patternrecognition technologies. It may be analyzed which of patternscorresponding to a classification prepared in advance (fruit, branch,leaf, etc.) matches the subject, so as to determine which class thesubject is. Pattern recognition is performed by use of, for example, atrainable pattern recognition system using mathematical, geometrical andharmonic-shape descriptors. Such a system is educated so as to select apossible class by use of a knowledge base learned by contacting a largenumber of samples of the classification prepared in advance. A trainingset includes thousands of basic types of images for each class.Successful pattern matching is caused when an untrained descriptor onthe target matches a trained descriptor on the target. Theabove-described pattern recognition system is described in detail in,for example, Japanese Laid-Open Patent Publication No. 2015-046162.

The mapping image shown in portion (c) of FIG. 9A includes mappingimages of three hierarchical layers I, II and III. For example, themapping image of the hierarchical layer I includes a distribution ofpixels having a gray scale value of 60 or larger and smaller than 70among the gray scale values of 0 to 255. The mapping image of thehierarchical layer II includes a distribution of pixels having a grayscale value of 70 or larger and smaller than 80 among the gray scalevalues of 0 to 255. The mapping image of the hierarchical layer IIIincludes a distribution of pixels having a gray scale value of 80 orlarger among the gray scale values of 0 to 255. The calculation deviceP, for example, maps the pixel distribution with blue to generate a bluemapping image of hierarchical layer I, maps the pixel distribution withgreen to generate a green mapping image of hierarchical layer II, andmaps the pixel distribution with red to generate a red mapping image ofhierarchical layer III. The calculation device P places the mappingimages of the three hierarchical layers I, II and III such that theimages overlap each other to generate the mapping image. However, thecalculation device P does not need to generate the color mapping imagesof the three hierarchical layers I, II and III. The calculation device Pmerely needs to generate at least a color mapping image of thehierarchical layer III based on one threshold value (e.g., “80”). Thenumber of the hierarchical layers may be determined in accordance with,for example, the number of maturity levels.

The calculation device P also generates the mapping image shown inportion (c) of FIG. 9B on the underripe bunch F and the mapping imageshown in portion (c) of FIG. 9C on the ripe bunch F, like the mappingimage on the unripe bunch F. The mapping image shown in portion (c) ofFIG. 9B is generated based on the pixel value NIRS in the image I shownin portion (b) of FIG. 9B, and includes color mapping imagesrespectively of the three hierarchical layers I, II and III. The mappingimage shown in portion (c) of FIG. 9C is generated based on the pixelvalue NIRS in the image I shown in portion (b) of FIG. 9C, and includescolor mapping images respectively of the three hierarchical layers I, IIand III.

As the maturity level of the bunch F rises, the intensity of thereflected light of the near infrared light, namely, the value of thepixel value NIRS increases. As a result, the density of the pixels ofthe hierarchical layer III is raised. This indicates that the maturitylevel of the bunch F is easily determinable by appropriately setting thethreshold value for the gray scale value.

Referring to FIG. 10, the three mapping images on the unripe, underripeand ripe bunch F are each divided into the three hierarchical layers I,II and III. FIG. 10 shows the color mapping images of the respectivehierarchical layers. As shown in FIG. 10, regarding the color mappingimages of the hierarchical layer I, the density of the pixels (pixeldistribution) is highest in the color mapping image on the unripe bunchF and lowest in the color mapping image on the ripe bunch F. Regardingthe color mapping images of the hierarchical layer II, the density ofthe pixels is highest in the color mapping image on the underripe bunchF and lowest in the color mapping image on the unripe bunch F. Regardingthe color mapping images of the hierarchical layer III, the density ofthe pixels is highest in the color mapping image on the ripe bunch F andlowest in the color mapping image on the unripe bunch F.

The pixel distribution corresponds to the area size ratio AR of theintensity distribution of the near infrared light on the basis of thepredetermined reference value. Specifically, the area size ratio AR isthe ratio of the number N1 of pixels fulfilling a condition for thereference value among the plurality of first pixels included in theimage I with respect to the number M1 of the plurality of first pixels.For example, the lower limit of the reference value (lower-limitthreshold value) and the upper limit of the reference value (upper-limitthreshold value) used for the hierarchical layer I are respectively 60and 70. The lower limit and the upper limit of the reference value usedfor the hierarchical layer II are respectively 70 and 80. The lowerlimit of the reference value used for the hierarchical layer III is 80.The upper limit is not set. An appropriate reference value may be set,so that the maturity level of the bunch F is determined based on thedensity of the pixels, namely, the area size ratio AR.

FIG. 11 is a graph obtained as a result of performing normal probabilityplotting of the signal strength of the near infrared light (i.e., pixelvalue NIRS) reflected by each of the unripe bunch F, the underripe bunchF and the ripe bunches F. The normal probability plot visuallyrepresents how close the sample data is to the normal distribution. Inthe normal probability plot, data conforming to the normal distributionis arranged linearly. From this, parameters such as the average value,the standard deviation, the representative value and the like may beestimated. The sample data in this embodiment is the pixel value NIRS ofthe plurality of first pixels included in the image I. The horizontalaxis of the graph represents the pixel value (value of 0 to 255), andthe vertical axis represents the cumulative frequency percentage (%)represented by logarithm.

The cumulative frequency percentage (%) is found from expression (1).

Cumulative frequency percentage (%)=(100×cumulativefrequency)/(n+1)  expression (1)

In expression (1), the cumulative frequency corresponds to a valueobtained by accumulating the frequency from the smallest value of thefrequency, namely, the left side of the histogram. n is the number ofsamples, and is equal to the number M1 of the plurality of first pixels.

As shown in FIG. 11, independent normal probability plots are obtainedfor the respective maturity levels. It is seen that as the maturitylevel of the bunch F rises, the normal probability plot is shiftedrightward, and as a result, the pixel value NIRS for a cumulativefrequency percentage of 50% increases. The pixel value at the cumulativefrequency percentage of 50% is a representative value of the pixel valueNIRS. In the normal distribution, the average value is equal to therepresentative value. The normal probability plot is effective fordetermining the above-described threshold values as the upper limit andthe lower limit. For example, these threshold values may be determinedbased on the representative value by referring to the normal probabilityplot.

FIG. 12 shows a specific processing procedure in step S200 in detail.The calculation device P finds the area size AR of the intensitydistribution of the near infrared light on the basis of thepredetermined reference value. Specifically, in step S210, thecalculation device P calculates the number N1 of pixels having a pixelvalue larger than, or equal to, a threshold value L1 from the pluralityof first pixels included in the image I. For example, for thehierarchical layer III shown in FIG. 10, “80” may be set as thethreshold value L1. As a result of computation, the number of pixelshaving a pixel value of “80” or larger is N1. Alternatively, N1 may bethe number of pixels having a pixel value smaller than, or equal to, thethreshold value L1, and in step S210, the calculation device P maycalculate the number N1 from the plurality of first pixels included inthe image I.

Next, in step S220, the calculation device P finds the ratio N1/M1,namely, the ratio the calculated number N1 of the pixels with respect tothe number M1 of the plurality of first pixels. It is now assumed thatthe image I is formed of only the plurality of first pixels. In thiscase, if the angle of view is narrow, the shadow between the fruits maypossibly influence the maturity determination. Therefore, it ispreferable that another threshold value is provided, so that the numberof first pixels having a pixel value larger than, or equal to, theanother threshold value (e.g., “10”) is set as M1 to find the ratioN1/M1.

Now, FIG. 6 is referred to again.

(Step S300)

The calculation device P generates maturity determination information inaccordance with the area size ratio AR. Specifically, the calculationdevice P generates the maturity determination information in accordancewith a result of comparing the ratio N1/M1 against a threshold value R1.

FIG. 13A shows a specific processing procedure in step S300 in detail.First, a procedure of determining whether the maturity level of thebunch F is “ripe” or not will be described. In step S310, thecalculation device P determines whether the ratio N1/M1 calculated forthe hierarchical layer III is larger than the threshold value R1 or not.The threshold value R1 is appropriately determined by the designingspecifications or the like, and is, for example, stored in advance in aninternal ROM (not shown) of the calculation device P. All the thresholdvalues described in this specification, including the threshold valueR1, are stored in the internal ROM. In the case where the ratio N1/M1 islarger than the threshold value R1, in step S320, the calculation deviceP generates maturity determination information representing “ripe”. Inthe case where the ratio N1/M1 is smaller than, or equal to, thethreshold value R1, the calculation device P generates maturitydetermination information representing, for example, “not ripe”. Thematurity determination information may be represented by, for example, a1-bit signal, where “0” may be assigned to “not ripe” and “1” may beassigned to “ripe”. With the above-described procedure, the maturitydetermination information representing whether the maturity level of thebunch F is “ripe” or not is obtained.

For example, it may be further determined whether the maturity level is“underripe” or “unripe”. Referring to FIG. 13A, in the case where theratio N1/M1 is determined to be smaller than, or equal to, the thresholdvalue R1 in step S310, it is at least known that the maturity level isnot “ripe”. In step S330, the calculation device P finds the ratio N1/M1for the hierarchical layer II and compares the ratio against a thresholdvalue R2.

Specifically, for the hierarchical layer II, the calculation device Pcalculates, among the plurality of first pixels, the number N1 of pixelshaving a pixel value larger than, or equal to, the lower limit of athreshold value L2 and smaller than the upper limit of the thresholdvalue L2. As described above, for example, the lower limit of thethreshold value L2 may be set to “70” and the upper limit of thethreshold value L2 may be set to “80”. Like in step S220, for thehierarchical layer II, the calculation device P finds the ratio N1/M1,namely, the ratio the calculated number N1 of the pixels with respect tothe number M1 of the plurality of first pixels.

In step S330, the calculation device P determines whether the ratioN1/M1 calculated for the hierarchical layer II is larger than thethreshold value R2 or not. In the case where the ratio N1/M1 is largerthan the threshold value R2, the calculation device P generates maturitydetermination information representing “underripe” in step S340. In thecase where the ratio N1/M1 is smaller than, or equal to, the thresholdvalue R2, the procedure goes to step S350.

In step 350, the calculation device P finds the ratio N1/M1 for thehierarchical layer I and compares the ratio against a threshold valueR3. Specifically, for the hierarchical layer I, the calculation device Pcalculates, among the plurality of first pixels, the number N1 of pixelshaving a pixel value larger than, or equal to, the lower limit of athreshold value L3 and smaller than the upper limit of the thresholdvalue L3. As described above, for example, the lower limit of thethreshold value L3 may be set to “60” and the upper limit of thethreshold value L3 may be set to “70”. Like in step S220, for thehierarchical layer I, the calculation device P finds the ratio N1/M1,namely, the ratio the calculated number N1 of the pixels with respect tothe number M1 of the plurality of first pixels.

In step S350, the calculation device P determines whether the ratioN1/M1 calculated for the hierarchical layer I is larger than thethreshold value R3 or not. In the case where the ratio N1/M1 is largerthan the threshold value R3, the calculation device P generates maturitydetermination information representing “unripe” in step S360. In thecase where the ratio N1/M1 is smaller than, or equal to, the thresholdvalue R3, the calculation device P generates maturity informationrepresenting, for example, “other” in step S370. The maturitydetermination information may be represented by, for example, a 2-bitsignal, where “11” may be assigned to “ripe”, “10” may be assigned to“underripe”, “01” may be assigned to “unripe”, and “00” may be assignedto “other”. In this manner, the maturity determination informationrepresenting “underripe” or “unripe” may be generated in addition to thematurity determination information representing “ripe”.

FIG. 13B shows another specific processing procedure in step S300 indetail. As shown in FIG. 13B, in the case where the ratio N1/M1 for thehierarchical layer III is larger than the threshold value R1 and theratio N1/M1 for the hierarchical layer II is larger than the thresholdvalue R2, the calculation device P can generate maturity determinationinformation representing “ripe”. In this manner, the maturitydetermination information can be generated using a plurality ofhierarchical layers.

Now, FIG. 6 is referred to again.

(Step S400)

In accordance with the maturity determination information, thecalculation device P generates a driving signal for driving thenotification device 500 notifying, for example, the farmer of thematurity level of the bunch F. For example, the notification device 500may include an LED. In such a structure, in the case where the maturitydetermination information represents “ripe”, the calculation device Pgenerates a driving signal for causing the LED to emit light (signal forturning on the LED). In the case where the maturity determinationinformation represents “not ripe”, the calculation device P does notgenerate a driving signal for causing the LED to emit light. Thenotification device 500 may include a plurality of LEDs emitting lightof different colors. In the case where the maturity determinationinformation includes various maturity levels (e.g., “ripe”, “underripe”and “unripe”), the calculation device P, for example, may generate adriving signal for causing any one of the plurality of LEDs to emitlight in accordance with the maturity level. In this manner, the farmerrecognizes the maturity level of the bunch F in accordance with thecolor of the light emitted by the LED.

For example, the notification device 500 may include a speaker. In sucha structure, in the case where the maturity determination informationrepresents “ripe”, the calculation device P generates a driving signalfor causing the speaker to output an audio signal (signal for turning onthe speaker). In the case where the maturity determination informationrepresents “not ripe”, the calculation device P does not generate adriving signal for causing the speaker to output an audio signal. In thecase where the maturity determination information includes variousmaturity levels, the calculation device P, for example, may drive thespeaker such that the loudness of the audio signal is changed inaccordance with the maturity level.

For example, the notification device 500 may include a vibrator. In sucha structure, in the case where the maturity determination informationrepresents “ripe”, the calculation device P generates a driving signalfor causing the vibrator to vibrate (signal for turning on thevibrator). In the case where the maturity determination informationrepresents “not ripe”, the calculation device P does not generate adriving signal for causing the vibrator to vibrate. In the case wherethe maturity determination information includes various maturity levels,the calculation device P, for example, may drive the vibrator such thatthe strong/weak pattern of the vibration is changed in accordance withthe maturity level.

For example, the notification device 500 may include a liquid crystaldisplay. In such a structure, in the case where the maturitydetermination information represents “ripe”, the calculation device Pgenerates a driving signal for causing the liquid crystal display todisplay letter information “ripe”. In the case where the maturitydetermination information represents “not ripe”, the calculation deviceP generates a driving signal for causing the liquid crystal display todisplay letter information “not ripe”. For example, the liquid crystaldisplay may display a symbol such as “◯”, “X” or the like instead of theletter information, or may change the color of display in accordancewith the maturity level.

The calculation device P may generate a color image of the bunch F basedon the pixel values REDS, GLNS and BLUS obtained from the RGB pixels ofthe image sensor 220. The calculation device P may generate a maturitylevel image including information on the reference value andrepresenting the maturity level, based on the maturity determinationinformation. The maturity level image is, for example, the mapping imageshown in portion (c) of FIG. 9A. The calculation device P may displaythe maturity level image, as overlapping the color image, on the liquidcrystal display. In this case, the calculation device P may also displayinformation on the threshold value used for the computation on theliquid crystal display. In this case, the farmer easily checks theactual position in the color image in relation with the mapping image.Thus, the harvest efficiency is improved.

The calculation device P may also determine whether the bunch F isharvestable or not based on the maturity determination information. Forexample, it is now assumed that there are four maturity levels of“ripe”, “slightly ripe”, “underripe” and “unripe”. In the case where,for example, the maturity determination information represents “ripe” or“slightly ripe”, the calculation device P determines that the bunch F isharvestable. In the case where the maturity determination informationrepresents “underripe” or “unripe”, the calculation device P determinesthat the bunch F is not harvestable. For example, the liquid crystaldisplay may display letter information representing whether the bunch Fis harvestable or not in accordance with the determination information.With such display, the information on whether the bunch F is harvestableor not is transmitted to the farmer directly, which supports the farmermore.

The calculation device P may output the maturity determinationinformation to outside via, for example, an output IF (not shown)instead of performing the operation in step S400. In the case where, forexample, the notification device 500 has a function of generating asignal for driving the notification device 500 itself, the notificationdevice 500 may receive the maturity determination information from thecalculation device P and generate a driving signal in accordance withthe maturity determination information.

In this embodiment, the variance in the determination on the maturitylevel at the time of harvest of oil palm is made smaller, regardlesswhether the farmer is experienced or not, than by a conventional methodby which the maturity level is determined by visual observation. Inaddition, the time to harvest the oil palm is determined with highprecision without decreasing the efficiency of the harvesting work.Therefore, it is expected that the production amount of the oilcomponent contained in the oil palm is increased.

Embodiment 2

According to a determination method in this embodiment, the calculationdevice P receives an image I including the intensity distributions ofthe light in the first wavelength band and the light of a secondwavelength band in step S100, and generate maturity determinationinformation based on the intensity distributions of light in the firstand second wavelength bands in step S300. The intensity of light of thesecond wavelength band that is reflected by the bunch F varies inaccordance with the maturity level. The second wavelength band isdifferent from the first wavelength band (i.e., the wavelength band ofthe near infrared light). The second wavelength band is a wavelengthband of, for example, red light, which is visible light.

FIG. 14A is a graph obtained as a result of performing normalprobability plotting of the signal strength of the red light (i.e.,pixel value REDS) reflected by each of the unripe bunch F, the underripebunch F and the ripe bunches F. FIG. 14B is a graph obtained as a resultof performing normal probability plotting of the pixel value that isobtained as a result of adding the pixel value NIRS and the pixel valueREDS. As in FIG. 11, the horizontal axis of the graph represents thepixel value (value of 0 to 255), and the vertical axis represents thecumulative frequency percentage (%) represented by logarithm.

The “addition of the pixel values” refers to adding pixel values ofpixels associated with each other in the pixel unit. In this embodiment,the calculation device P adds the pixel value of the pixel 221Cincluding the IR filter and the pixel value of the pixel 221A includingthe R filter that is associated with the pixel 221C. In other words, thecalculation device P adds the pixel value NIRS and the pixel value REDS.

As shown in FIG. 14B, like in FIG. 11, independent normal probabilityplots are obtained for the respective maturity levels. As compared within the normal probability plots based on the pixel value NIRS shown inFIG. 11, in the normal probability plots based on the sum of the pixelvalue NIRS and the pixel value REDS shown in FIG. 14B, the pixel valueat a cumulative frequency percentage of 50% (i.e., the representativevalue) is larger.

FIG. 15 shows color mapping images obtained as a result of dividing eachof the three mapping images on the unripe, underripe and ripe bunch Finto the three hierarchical layers I, II and III, like in FIG. 10. Therectangular parts of the unripe, underripe and ripe bunch F shown inportions (a) of FIG. 9A, FIG. 9B and FIG. 9C are used in FIG. 15 for thematurity determination. The mapping images shown in FIG. 15 aregenerated based on the sum of the pixel value NIRS and the pixel valueof REDS in the image I.

As shown in FIG. 15, regarding the color mapping images of thehierarchical layer I, the density of the pixels (pixel distribution) ishighest in the color mapping image on the unripe bunch F and lowest inthe color mapping image on the ripe bunch F. Regarding the color mappingimages of the hierarchical layer II, the density of the pixels ishighest in the color mapping image on the underripe bunch F and lowestin the color mapping image on the unripe bunch F. Regarding the colormapping images of the hierarchical layer III, the density of the pixelsis highest in the color mapping image on the ripe bunch F and lowest inthe color mapping image on the unripe bunch F. What should be paidattention is that as compared with in the FIG. 10, the difference in thepixel distribution (density) between the color mapping image on theunderripe bunch F and the color mapping image on the ripe bunch F at thehierarchical layer III is larger. This indicates that the precision ofthe maturity determination may be improved by setting an appropriatethreshold value between the slightly underripe bunch F and the ripebunch F.

FIG. 16 shows a specific processing procedure in step S200 in detail. Instep S230, the calculation device P adds the pixel value NIRS of eachfirst pixel of the plurality of first pixels and the pixel value REDS ofeach second pixel of the plurality of second pixels, on a pixel-by-pixelbasis. Said each second pixel is associated with said each first pixel.Namely, the calculation device P adds the pixel value NIRS and the pixelvalue REDS on a pixel unit-by-pixel unit basis. In step S240, thecalculation device P calculates, among the plurality of pixels that aretargets of addition, the number N2 of pixels having a sum value largerthan, or equal to, a threshold value L4. In step S250, the calculationdevice P finds the ratio N2/M2, namely, the ratio the calculated numberN2 of the pixels with respect to the number M2 of the pixels that arethe targets of addition. Typically, the number M2 of the pixels that arethe targets of addition matches the number of the plurality of firstpixels. For example, in the case where one pixel unit includes threepixels 221C including the IR filter and one pixel 221A including the Rfilter, the pixel value NIRS of each of the three pixels 221C is addedwith the pixel value REDS of the pixel 221A. Alternatively, N2 may bethe number of pixels having a sum value smaller than, or equal to, thethreshold value L4, and in step S240, the calculation device P maycalculate the number N2 from the pixels that are the targets ofsubtraction.

In this embodiment, the pixel distribution is represented by the ratioN2/M2, and corresponds to the area size ratio AR of the sum of theintensity distributions of the near infrared light and the red light onthe basis of the predetermined reference value. For example, the lowerlimit of the reference value (lower-limit threshold value) and the upperlimit of the reference value (upper-limit threshold value) used for thehierarchical layer I are respectively 90 and 100. The lower limit andthe upper limit of the reference value used for the hierarchical layerII are respectively 100 and 110. The lower limit of the reference valueused for the hierarchical layer III is 110. The upper limit is not set.

Next, the calculation device P follows the processing procedure shownin, for example, FIG. 13A to generate the maturity determinationinformation in accordance with a result of comparing the ratio N2/M2against a threshold value R4. Specifically, like in step S310, thecalculation device P determines which of the ratio N2/M2 for thehierarchical layer III and the threshold value R4 is larger or smaller.In the case where the ratio N2/M2 is larger than threshold value R4, thecalculation device P generates maturity determination informationrepresenting “ripe” like in step S320.

With the determination method in this embodiment, the information on thepixel value REDS is used in addition to the information on the pixelvalue NIRS, so that the precision of the maturity determination isfurther improved.

Embodiment 3

According to a determination method in this embodiment, unlike thedetermination method in embodiment 2, maturity determination informationis generated based on a difference between the pixel NIRS and the pixelvalue REDS.

For example, the light reflected by the bunch F includes light reflectedby the fruits and also light reflected by the dead calyx and leavescovering the fruits. The dead calyx and leaves contains littlechlorophyll, which is a main component of chloroplast. Therefore, thereflectance characteristic of the light reflected by the dead calyx andleaves is generally uniform in a wavelength band from the red light tothe near infrared light. By contrast, as shown in FIG. 7, the lightreflected by the bunch F has a reflectance characteristic that thereflectance is especially high in a part of the wavelength from the redlight to the near infrared light. The image I acquired by the imagecapturing device 200 may include the dead calyx and leaves. Therefore,the image I used for the determination method in each of embodiments 1and 2 may include complicated information caused by the dead calyx andleaves. In this embodiment, the above-described difference between thereflectance characteristic of the light reflected by the dead calyx andleaves and the reflectance characteristic of the light reflected by thefruits is paid attention to, and a difference between an image A formedof the pixel value NIRS and an image B formed of the pixel value REDS isfound. The image A includes the dead calyx and leaves, and the image Bmainly includes the dead calyx and leaves. The difference between theimage A and the image B is found, so that information mainly caused bythe fruits is selectively obtained.

FIG. 17 is a graph obtained as a result of performing normal probabilityplotting of the pixel value that is obtained as a result of subtractingthe pixel value REDS from the pixel value NIRS. As in FIG. 11, thehorizontal axis of the graph represents the pixel value (value of 0 to255), and the vertical axis represents the cumulative frequencypercentage (%) represented by logarithm.

The “subtraction of the pixel values” refers to subtracting the pixelvalue of a pixel from the pixel value of another pixel associated withthe above-mentioned pixel in the pixel unit. In this embodiment, thecalculation device P subtracts, from the pixel value of the pixel 221Cincluding the IR filter, the pixel value of the pixel 221A including theR filter that is associated with the pixel 221C. In other words, thecalculation device P subtracts the pixel value REDS from and the pixelvalue NIRS.

As shown in FIG. 17, like in FIG. 11, independent normal probabilityplots are obtained for the respective maturity levels. As compared withthe normal probability plots based on the sum of the pixel values shownin FIG. 14B, the normal probability plots based on the differencebetween the pixel values shown in FIG. 17 are more straight, which showsthat the characteristic is closer to the normal distribution.

FIG. 18 shows color mapping images obtained as a result of dividing eachof the three mapping images on the unripe, underripe and ripe bunch Finto the three hierarchical layers I, II and III, like in FIG. 10. Therectangular parts of the unripe, underripe and ripe bunch F shown inportions (a) of FIG. 9A, FIG. 9B and FIG. 9C are used in FIG. 18 for thematurity determination. The mapping images shown in FIG. 18 aregenerated based on the difference between the pixel value NIRS and thepixel value REDS in the image I.

As shown in FIG. 18, regarding the color mapping images of thehierarchical layer I, the density of the pixels (pixel distribution) ishighest in the color mapping image on the unripe bunch F and lowest inthe color mapping image on the ripe bunch F. Regarding the color mappingimages of the hierarchical layer II, the density of the pixels ishighest in the color mapping image on the underripe bunch F and lowestin the color mapping image on the unripe bunch F. Regarding the colormapping images of the hierarchical layer III, the density of the pixelsis highest in the color mapping image on the ripe bunch F and lowest inthe color mapping image on the unripe bunch F.

FIG. 19 shows a specific processing procedure in step S200 in detail. Instep S260, the calculation device P subtracts the pixel value REDS ofeach second pixel of the plurality of second pixels from the pixel valueNIRS of each first pixel of the plurality of first pixels, on apixel-by-pixel basis. Said each second pixel is associated with saideach first pixel. Namely, the calculation device P subtracts the pixelvalue REDS from the pixel value NIRS on a pixel unit-by-pixel unitbasis. In step S270, the calculation device P calculates, among theplurality of pixels that are targets of subtraction, the number N3 ofpixels having a difference value larger than, or equal to, a thresholdvalue L5. In step S280, the calculation device P finds the ratio N3/M3,namely, the ratio the calculated number N3 of the pixels with respect tothe number M3 of the pixels that are the targets of subtraction.Typically, the number M3 of the pixels that are the targets ofsubtraction matches the number of the plurality of first pixels. Forexample, in the case where one pixel unit includes three pixels 221Cincluding the IR filter and one pixel 221A including the R filter, thepixel value REDS of the pixel 221A is subtracted from the pixel valueNIRS of each of the three pixels 221C. Alternatively, N3 may be thenumber of pixels having a difference value smaller than, or equal to,the threshold value L5, and in step S270, the calculation device P maycalculate the number N3 from the pixels that are the targets ofsubtraction.

In this embodiment, the pixel distribution is represented by the ratioN3/M3, and corresponds to the area size ratio AR of the differencebetween the intensity distributions of the near infrared light and thered light on the basis of the predetermined reference value. Forexample, the lower limit of the reference value (lower-limit thresholdvalue) and the upper limit of the reference value (upper-limit thresholdvalue) used for the hierarchical layer I are respectively 40 and 50. Thelower limit and the upper limit of the reference value used for thehierarchical layer II are respectively 50 and 60. The lower limit of thereference value used for the hierarchical layer III is 60. The upperlimit is not set.

Next, the calculation device P follows the processing procedure shownin, for example, FIG. 13A to generate the maturity determinationinformation in accordance with a result of comparing the ratio N3/M3against a threshold value R5. Specifically, like in step S310, thecalculation device P determines which of the ratio N3/M3 for thehierarchical layer III and the threshold value R5 is larger or smaller.In the case where the ratio N3/M3 is larger than threshold value R5, thecalculation device P generates maturity determination informationrepresenting “ripe” like in step S320.

With the determination method in this embodiment, the informationobtained as a result of subtracting the pixel value REDS from the pixelvalue NIRS is used, so that the precision of the maturity determinationis further improved while the complicated information included in theimage I is suppressed.

Embodiment 4

In this embodiment, the calculation device P finds the number N4 ofpixels having a pixel value larger than, or equal to, a threshold valueL6 or a pixel value smaller than, or equal to, the threshold value L6from the plurality of second pixels, and finds the ratio N4/M4, namely,the ratio the calculated number N4 of the pixels with respect to thenumber M4 of the plurality of second pixels. The calculation device Pfurther generates maturity determination information based on a resultof comparing the ratio N1/M1 against the threshold value R1 and a resultof comparing the ratio N4/M4 against a threshold value R6.

FIG. 20 shows a specific processing procedure in step S200 in detail.

First, the calculation device P generates a color mapping image of thehierarchical layer III by use of the plurality of first pixels includedin the image I. Specifically, the calculation device P calculates thenumber N1 of the pixels by use of the color mapping image of thehierarchical layer III in step S210, and finds the ratio N1/M1 in stepS220. Next, the calculation device P generates a color mapping image ofthe hierarchical layer III by use of the plurality of second pixelsincluded in the image I. Specifically, in step S211, the calculationdevice P calculates the number N4 of pixels having a pixel value largerthan, or equal to, a threshold value L6 from the plurality of secondpixels included in the image I. For example, for the hierarchical layerIII shown in FIG. 10, “80” may be set as the threshold value L6. As aresult of computation, the number of pixels having a pixel value of “80”or larger is N4. Alternatively, N4 may be the number of pixels having apixel value smaller than, or equal to, the threshold value L6, and instep S211, the calculation device P may calculate the number N4 from theplurality of second pixels included in the image I.

Next, in step S221, the calculation device P finds the ratio N4/M4,namely, the ratio the calculated number N4 of the pixels with respect tothe number M4 of the plurality of second pixels. It is now assumed thatthe image I is formed of only the plurality of first pixels. In thiscase, if the angle of view is narrow, the shadow between the fruits maypossibly influence the maturity determination. Therefore, it ispreferable that another threshold value is provided for the plurality ofsecond pixels, so that the number of second pixels having a pixel valuelarger than, or equal to, the another threshold value (e.g., “10”) isset as M4 to find the ratio N4/M4.

Next, in step S310, the calculation device P checks which of the ratioN1/M1 for the hierarchical layer III based on the plurality of firstpixels and the threshold value R5 is larger or smaller. The calculationdevice P further checks which of the ratio N4/M4 for the hierarchicallayer III based on the plurality of second pixels and the thresholdvalue R6 is larger or smaller. In the case where the ratio N1/M1 islarger than threshold value R1 and the ratio N4/M4 is larger thanthreshold value R6, the calculation device P generates maturitydetermination information representing “ripe” like in step S320.

With the determination method in this embodiment, in the case where thebunch F cannot be selected as being distinguished from another object, asensor condition for the pixel value NIRS and a sensor condition for thepixel value REDS are independently set, so that the selectivity of thetarget is improved.

Embodiment 5

With reference to FIG. 21 through FIG. 24, a structure of a maturitydetermination device 100 in this embodiment will be mainly described.The maturity determination device 100 is operable in accordance with theprocessing procedure of the determination method described in each ofembodiments 1 through 4.

FIG. 21 schematically shows a block structure of the maturitydetermination device 100. The maturity determination device 100typically includes a controller C, an image capturing device 200, asignal calculation device 300 and a driving circuit 400. The maturitydetermination device 100 is connectable with the notification device 500in a wired or wireless manner. For example, the maturity determinationdevice 100 may include a USB interface (not shown) and may be connectedwith the notification device 500 via a USB cable.

The controller C is a semiconductor integrated circuit (LSI), and maybe, for example, a general-purpose processor. The controller C iselectrically connected with each of the image capturing device 200, thesignal processing circuit 300 and the driving circuit 400 to control theentirety of the maturity determination device 100.

The structure of the image capturing device 200 is as described above.As shown in FIG. 2, the image capturing device 200 includes theplurality of pixels 221 arrayed one-dimensionally or two-dimensionally,and performs image capturing of at least a part of the bunch F toacquire the image I. The pixel array 220A is not limited to having theshape shown in FIG. 2, and may have, for example, the shape shown FIG.5.

FIG. 22 schematically shows functional blocks of the signal processingcircuit 300. Each of the functional blocks of the signal processingcircuit 300 is represented on a functional block unit basis, not on ahardware unit basis.

The signal processing circuit 300 is a semiconductor integrated circuit,and is, for example, an image signal processor (ISP). The signalprocessing circuit of the calculation device P described in embodiment 1is an element corresponding to the signal processing circuit 300. Thesignal processing circuit 300 includes an area size ratio computationsection 310, a maturity determination information generation section320, a driving signal generation section 330, a storage medium 340 and acentral processing section (CPU) 350. The storage medium 340 is, forexample, a read-only memory (ROM), a random access memory (RAM), whichis writable, or a hard disc. The area size ratio computation section310, the maturity determination information generation section 320, andthe driving signal generation section 330 may be realized by anycombination of hardware and software using, for example, the CPU 350,the RAM (not shown), a computer program loaded on the RAM, the ROM(corresponding to the storage medium 340) storing the computer program,and an interface for connection with a network.

The signal processing circuit 300 includes hardware and/or software, andis configured to operate in accordance with the processing procedure ofthe determination method described in each of embodiments 1 through 4.For example, the calculation device P (specifically, the signalprocessing circuit) as the operating subject of the processingprocedures of the determination methods shown in FIG. 6, FIG. 12, FIGS.13A and 13B may be replaced with the signal processing circuit 300, sothat a specific operation of the signal processing circuit 300 isrealized. A computer program including a group of commands includingsuch a processing procedure is stored on, for example, the ROM 340. Inthis embodiment, a description of the operation of the signal processingcircuit 300 will be omitted.

Like the calculation device P, the signal processing circuit 300 maygenerate the color image I of the bunch F based on the pixel valuesREDS, GLNS and BLUS obtained from the RGB pixels of the image sensor220. The signal processing circuit 300 may also generate a maturitylevel image, including information on the reference value andrepresenting the maturity level, based on the maturity determinationinformation. The signal processing circuit 300 may overlap the maturitylevel image on the color image to generate an overlapping image to bedisplayed on the notification device 500 (e.g., liquid crystal display).The signal processing circuit 300 may further determine whether thebunch F is harvestable or not based on the maturity determinationinformation.

The driving circuit 400 may include hardware and/or software. Thedriving circuit 400 is, for example, a semiconductor integrated circuit.The driving circuit 400 generates a driving signal in accordance withthe maturity determination information or the determination informationon whether the bunch F is harvestable or not in step S400 in theprocessing procedure shown in FIG. 6. An example of generating a drivingsignal is described in embodiment 1 in detail.

A part of the maturity determination device 100 may be realized by anLSI as one chip. Each of the functional blocks of the signal processingcircuit 300 may be independently a chip. Alternatively, a part of, orthe entirety of, the functional blocks of the signal processing circuit300 may be integrated into a chip. The semiconductor integrated circuitmay be, for example, an application specific integrated circuit (ASIC)or a field programmable gate array (FPGA). In the case where atechnology of an integrated circuit usable as a substitute of the LSIemerges by the advancement of the semiconductor technology, anintegrated circuit realized by such a technology is usable.

FIG. 23 schematically shows a block structure of a maturitydetermination device 100A in a variation of this embodiment. FIG. 24schematically shows a block structure of a maturity determination device100B in another variation of this embodiment.

The maturity determination device 100A includes the controller C, theimage capturing device 200, the signal processing circuit 300 and anoutput interface (IF) 600. The maturity determination device 100Aincludes the output IF 600 instead of the driving circuit 400. Thematurity determination device 100A may output the maturity determinationinformation to outside via the output IF 600. The output IF 600 is a USBinterface or an interface configured to perform a wireless communicationconformed to, for example, the Bluetooth standards. In this variation,in the case where, for example, the notification device 500 has afunction of generating a signal for driving the notification device 500itself, the notification device 500 may receive the maturitydetermination information from the signal processing circuit 300 andgenerate a driving signal in accordance with the maturity determinationinformation. The maturity determination device 100A may be provided witha driving circuit as an external element. The driving circuit mayreceive the maturity determination information and may generate adriving signal in accordance with the maturity determinationinformation.

The maturity determination device 100B includes the controller C, theimage capturing device 200, the signal processing circuit 300, thedriving circuit 400 and the notification device 500. In this variation,the notification device 500 is integrally included in the maturitydetermination device 100B. The maturity determination device 100B isdecreased in size and thus is easier to use for the farmer.

The signal processing circuit 300 may have a plurality of operationmodes respectively operable in accordance with the determination methodin each of embodiments 1 through 4. For example, the signal processingcircuit 300 may select an optimal operation mode, from the plurality ofoperation modes, for switching the sensor sensitivity of the imagesensor 220 or for narrowing down the sensing results.

In embodiments 1 through 5 described above, the determination method isdescribed regarding an oil palm bunch as an example of fruit orvegetable product, which is a harvesting target. The fruit or vegetableproduct targeted by the present invention is not limited to the oil palmbunch, and may be any fruit or vegetable product having a reflectancecharacteristic of the near infrared light that the reflectance varies inaccordance with the maturity level. An example of such a fruit orvegetable product is, for example, green apple or mango.

FIG. 25A shows an example of wavelength dependence of the reflectance oflight by green apple. FIG. 25B shows an example of wavelength dependenceof the reflectance of light by mango. The horizontal axis represents thewavelength (nm) of the light, and the vertical axis represents thereflectance (%). Like in the case of the oil palm bunch F, thereflectance tends to increase as the maturity level rises especially inthe wavelength band of the red light. It is seen that also in thewavelength band of the near infrared light, the reflectance tends toincrease as the maturity level rises. Therefore, an embodiment of thepresent invention is preferably applicable to green apple and mango.Substantially the same effects are provided as for the oil palm bunch F.The target of determination of an embodiment of the present invention isnot limited to the maturity level, and may be any of other indexes bywhich the time to harvest may be learned (e.g., growing degree,freshness, harvest level, etc.).

The maturity determination information may be generated based on an NDVI(Normalized Difference Vegetation Index). The NDVI is generated based onthe pixel value NIRS and the pixel value REDS, as found from expression(2).

(NIRS−REDS)/(NIRS+REDS)  (2)

The unripe bunch F contains more chlorophyll than does the ripe bunch F.Chlorophyll absorbs red light well and reflects near infrared lightwell. Therefore, the maturity level of the bunch F may well bedetermined by using the NDVI.

The calculation device P calculates the NDVI using the pixel value NIRSof each first pixel of the plurality of first pixels and the pixel valueREDS of each second pixel of the plurality of second pixels, on apixel-by-pixel basis. Said each second pixel is associated with saideach first pixel. Namely, the calculation device P calculates the NDVIusing the pixel value NIRS and the pixel value REDS on a pixelunit-by-pixel unit basis. The calculation device P calculates, among theplurality of pixels that are targets of calculation, the number N100 ofpixels having an NDVI value larger than, or equal to, a threshold valueL100. The calculation device P finds the ratio N100/M100, namely, theratio of the calculated number N100 of the pixels with respect to thenumber M100 of pixels that are the targets of calculation. Typically,the number M100 of pixels that are the targets of calculation matchesthe number of the plurality of first pixels.

The pixel distribution is represented by the ratio N100/M100, andcorresponds to the area size ratio AR of the NDVI value on the basis ofthe predetermined reference value. For example, the lower limit of thereference value (lower-limit threshold value) and the upper limit of thereference value (upper-limit threshold value) used for the hierarchicallayer I are respectively 90 and 100. The lower limit and the upper limitof the reference value used for the hierarchical layer II arerespectively 100 and 110. The lower limit of the reference value usedfor the hierarchical layer III is 110. The upper limit is not set.

The maturity determination information is generated in accordance with aresult of comparing the ratio N100/M100 against a threshold value R100.Specifically, the calculation device P determines which of the ratioN100/M100 for the hierarchical layer III and the threshold value R100 islarger or smaller. In the case where the ratio N100/M100 is larger thanthreshold value R100, the calculation device P generates maturitydetermination information representing “ripe”.

With the determination method, the NDVI information can be used tofurther improve the precision of the maturity determination.

A maturity determination device in an embodiment according to thepresent invention determining a maturity level of a fruit or vegetableproduct, comprising:

an image capturing device including a plurality of pixels arrayedone-dimensionally or two-dimensionally, the image capturing deviceperforming image capturing of at least a part of the fruit or vegetableproduct to acquire an image, the plurality of pixels including aplurality of first pixels each including a near infrared lighttransmission filter selectively transmitting light of a near infraredwavelength band and a plurality of second pixels each including a redlight transmission filter selectively transmitting light of redwavelength band; and

a signal processing circuit configured to find an area size ratio of anintensity distribution of NDVI on the basis of a predetermined referencevalue based on pixel values obtained from the plurality of first pixelsand pixel values obtained from the plurality of second pixels, and togenerate maturity determination information in accordance with the areasize ratio.

A method in an embodiment according to the present invention is a methodfor determining a maturity level of a fruit or vegetable product, themethod comprising the steps of:

receiving an image including at least a part of the fruit or vegetableproduct, the image including an intensity distribution of near infraredlight and an intensity distribution of red light;

finding an area size ratio of the intensity distribution of NDVI on thebasis of a predetermined reference value; and

generating maturity determination information in accordance with thearea size ratio.

The determination method in an embodiment according to the presentinvention may be realized by a computer program. The computer program isconfigured to realize the various functions described above inembodiments 1 through 5. The computer program controls, for example, thecalculation device P and the CPU or the like of the maturitydetermination device 100. Information handled by these devices istemporarily stored on the RAM when being processed, and then is storedon any of various ROMs or HDDs. The CPU reads the information whennecessary, and corrects the information or write additional data on theinformation. The storage medium storing the computer program may be, forexample, a semiconductor storage medium (e.g., ROM, nonvolatile memorycard, etc.), an optical storage medium (e.g., DVD, MO, MD, CD, BD,etc.), a magnetic storage medium (e.g., magnetic tape, flexible disc,etc.) or the like. Each of the functions described above in embodiments1 through 5 is realized by the CPU loading and executing the computerprogram. Alternatively, each of the functions described above inembodiments 1 through 5 may be realized by the CPU in cooperation withan operating system, another application program or the like inaccordance with an instruction of the computer program.

The computer program may be stored on a portable storage medium, so thatthe contents thereof are distributed in the market. Alternatively, thecomputer program may be transferred to a server computer connected via anetwork such as the Internet or the like, so that the contents thereofare distributed in the market. In this case, the storage device includedin the server computer is encompassed in the present invention. Each ofthe functions described above in embodiments 1 through 5 may be storedon a computer-readable medium, or transferred, as at least one commandgroup or a code. The “computer-readable storage medium” encompasses acommunication medium including a medium assisting the computer programbe carried from one site to another site and also encompasses a computerstorage medium. The storage medium may be any commercially availablemedium accessible by a general-purpose computer or a special-purposecomputer.

In this specification, various illustrative elements, blocks, modules,circuits and steps are described generally regarding the functionalitythereof in order to clearly show the synonymy between hardware andsoftware. Whether such functionality is implemented as hardware orsoftware depends on the designing restriction imposed on each ofapplications and the entire system. A person of ordinary skill in theart could implement the functions by any of various methods for specificapplications, but determination on such implementation should not beconstrued as departing from the scope of this disclosure.

Various illustrative logical blocks and processing units described inrelation with the disclosure of this specification may be implemented orexecuted by a general-purpose processor, a digital signal processor(DSP), an ASIC, an FPGA, any other programmable logical device, adiscrete gate or transistor logic, or a discrete hardware componentdesigned to execute the functions described in this specification, or acombination of any of these. The general-purpose processor may be amicroprocessor, or may be a conventional processor, controller,microcontroller or state machine. The processor may be implemented by acombination of computing devices. For example, the processor may berealized by a combination of a DSP and a microprocessor, a combinationof a plurality of microprocessors, a combination of a DSP core and atleast one microprocessor connected with the DSP core, or a combinationof any other such devices.

The determination methods or the steps of algorithm described inrelation with the disclosure of this specification may be directlyembodied by a software module executable by hardware (especially, aprocessor) or by a combination of hardware and the software module. Thesoftware module may be present in a RAM, a flash memory, a ROM, anEPROM, an EEPROM, a register, a hard disc, a removable disc, a CD-ROM,or any storage medium in any form known in this field. A typical storagemedium may be coupled to the processor such that information may be readtherefrom, or written thereto, by the processor. With another method,the storage medium may be integrated with the processor. The processorand the storage medium may be in the ASIC. The ASIC may be mounted onthe maturity determination device. Alternatively, the processor and thestorage medium may be housed in the maturity determination device asdiscrete elements.

Embodiments of the present invention have been described in detail withreference to the drawings. Specific structures are not limited to any ofthe embodiments, and designs and the like not departing from the gist ofthe present invention are encompassed in the scope of the claims.

A maturity determination method and a maturity determination device inan embodiment according to the present invention are preferably usablefor a method and a device determining the maturity level of a fruit orvegetable product, especially the maturity level of a bunch including alarge number of fruits.

While the present invention has been described with respect to preferredembodiments thereof, it will be apparent to those skilled in the artthat the disclosed invention may be modified in numerous ways and mayassume many embodiments other than those specifically described above.Accordingly, it is intended by the appended claims to cover allmodifications of the invention that fall within the true spirit andscope of the invention.

What is claimed is:
 1. A method for determining a maturity level of afruit or vegetable product, the method comprising the steps of:receiving an image including at least a part of the fruit or vegetableproduct, the image including an intensity distribution of light of atleast a first wavelength band reflected by the fruit or vegetableproduct and varying in accordance with the maturity level; finding anarea size ratio of the intensity distribution of the light of the firstwavelength band based on a predetermined reference value; and generatingmaturity determination information in accordance with the area sizeratio.
 2. The method according to claim 1, wherein the first wavelengthband is a wavelength band of near infrared light.
 3. The methodaccording to claim 2, wherein the first wavelength band is a wavelengthband of near infrared light from 800 nm to 900 nm.
 4. The methodaccording to claim 1, wherein: in the step of finding the area sizeratio, a number of pixels having a pixel value larger than, or equal to,a first threshold value or a pixel value smaller than, or equal to, thefirst threshold value is calculated from a plurality of first pixelsincluding information representing the intensity distribution of thelight of the first wavelength band and included in the image; and afirst ratio of the calculated number of the pixels with respect to anumber of the plurality of first pixels is determined; and in the stepof generating the maturity determination information, the maturitydetermination information is generated in accordance with a result ofcomparing the first ratio against a second threshold value.
 5. Themethod according to claim 1, wherein: in the step of receiving theimage, the image received further includes an intensity distribution oflight of a second wavelength band reflected by the fruit or vegetableproduct and varying in accordance with the maturity level, the secondwavelength band being different from the first wavelength band; and inthe step of generating the maturity determination information, thematurity determination information is generated based on the intensitydistributions of the light of the first wavelength band and the light ofthe second wavelength band.
 6. The method according to claim 5, whereinthe second wavelength band is a wavelength band of red light.
 7. Themethod according to claim 5, wherein: in the step of finding the areasize ratio, a pixel value of each of the plurality of first pixelsincluding information representing the intensity distribution of thelight of the first wavelength band and included in the image, and apixel value of each of the plurality of second pixels includinginformation representing the intensity distribution of the light of thesecond wavelength band and included in the image, are added together ona pixel-by-pixel basis, said each second pixel being associated withsaid each first pixel; a number of pixels having a sum value largerthan, or equal to, a third threshold value or a sum value smaller than,or equal to, the third threshold value is calculated from pixels astargets of addition; and a second ratio of the calculated number of thepixels with respect to the number of the pixels as the targets ofaddition is determined; and in the step of generating the maturitydetermination information, the maturity determination information isgenerated in accordance with a result of comparing the second ratio witha fourth threshold value.
 8. The method according to claim 5, wherein:in the step of finding the area size ratio, a pixel value of each of theplurality of second pixels including information representing theintensity distribution of the light of the second wavelength band andincluded in the image is subtracted, on a pixel-by-pixel basis, from apixel value of each first pixel of the plurality of first pixelsincluding information, representing the intensity distribution of thelight of the first wavelength band, and included in the image, said eachsecond pixel being associated with said each first pixel; a number ofpixels having a difference value larger than, or equal to, a thirdthreshold value or a difference value smaller than, or equal to, thethird threshold value is calculated from pixels as targets ofsubtraction; and a third ratio of the calculated number of the pixelswith respect to the number of the pixels as the targets of subtractionis determined; and in the step of generating the maturity determinationinformation, the maturity determination information is generated inaccordance with a result of comparing the third ratio against a fourththreshold value.
 9. The method according to claim 5, wherein: in thestep of finding the area size ratio, a number of pixels having a pixelvalue larger than, or equal to, a fifth threshold value or a pixel valuesmaller than, or equal to, the fifth threshold value is calculated fromthe plurality of second pixels including information representing theintensity distribution of the light of the second wavelength band andincluded in the image; a fourth ratio of the calculated number of thepixels with respect to the number of the plurality of second pixels isdetermined; and the maturity determination information is generated inaccordance with a result of comparing the first ratio against the secondthreshold value and a result of comparing the fourth ratio against asixth threshold value.
 10. The method according to claim 6, wherein inthe step of receiving the image, the image received further includesintensity distributions of blue light and green light; the methodfurther comprising generating a color image based on pixel valuesobtained from the plurality of pixels including information representingthe intensity distributions of the red light, the blue light and thegreen light, and generating a maturity level image including informationon the reference value and representing the maturity level, the maturitylevel image being generated based on the maturity determinationinformation.
 11. The method according to claim 1, further comprising thestep of outputting the maturity determination information to outside.12. The method according to claim 1, further comprising generating adriving signal to drive a notification device notifying the maturitylevel, the driving signal being generated in accordance with thematurity determination information.
 13. The method according to claim12, wherein the notification device includes at least one of an opticaldevice to emit light in accordance with the driving signal, a soundoutput device to output a sound in accordance with the driving signal, avibration device to vibrate in accordance with the driving signal, and adisplay device to display maturity level information in accordance withthe driving signal.
 14. The method according to claim 10, furthercomprising displaying the maturity level image as overlapping the colorimage on a display device.
 15. The method according to claim 1, furthercomprising performing image capturing of at least a portion of the fruitor vegetable product to acquire the image.
 16. The method according toclaim 15, wherein in the step of acquiring the image, the image isacquired by performing image capturing of a central portion and avicinity thereof of the fruit or vegetable product.
 17. The methodaccording to claim 15, wherein in the step of acquiring the image, theimage is acquired a plurality of times.
 18. The method according toclaim 1, further comprising determining whether the fruit or vegetableproduct is harvestable or not based on the maturity determinationinformation.
 19. The method according to claim 1, wherein the lightreflected by the fruit or vegetable product at a time of harvest thereofhas a reflectance characteristic with an intensity that increases as thelight has a longer wavelength in a wavelength band from a blue light toa near infrared light.
 20. The method according to claim 1, wherein thefruit or vegetable product is a bunch including a plurality of fruits.